Papers with Kendall correlation
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator (2025.coling-industry)
Copied to clipboard
| Challenge: | Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. |
| Approach: | They propose a framework that uses a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. |
| Outcome: | The proposed framework achieves high correlation with human judgment and a consistent rating and adaptability to custom error guidelines. |
Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have not investigated the differences between different correlation measures in meta-evaluation. |
| Approach: | They analyze 12 common correlation measures using real-world data from six widely-used NLG evaluation datasets and 32 evaluation metrics. |
| Outcome: | The proposed measures exhibit the best performance in discriminative power and ranking consistency . the measures using system-level grouping or Kendall correlation are the least sensitive to score granularity . |
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data (2022.acl-long)
Copied to clipboard
| Challenge: | Instance-level difficulty analysis of evaluation data is a new field of research that focuses on leveraging instance difficulty in natural language processing. |
| Approach: | They conduct Instance-Level Difficulty Analysis of Evaluation data in a large-scale setup of 23 datasets and demonstrate its five novel applications. |
| Outcome: | The proposed model improves efficiency and accuracy, improves quality and improves Out-of-Domain performance. |
FinGrAct: A Framework for FINe-GRrained Evaluation of ACTionability in Explainable Automatic Fact-Checking (2025.findings-emnlp)
Copied to clipboard
| Challenge: | despite the importance of actionability, no prior research has evaluated its effectiveness. |
| Approach: | They propose a fine-grained evaluation framework that can access the web to assess actionability in AFC explanations. |
| Outcome: | The proposed framework surpasses state-of-the-art evaluators in achieving highest correlation with human judgments while showing lowest egocentricbias. |
Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Traditional evaluation metrics rely heavily on lexical similarity with human-written references, showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences. |
| Approach: | They propose an interpretable evaluation framework that evaluates alignment with specific human preferences by providing detailed comments and fine-grained scoring. |
| Outcome: | The proposed framework outperforms GPT-4 in Kendall correlation and accuracy with zero-shot reviewers. |